Back to index

The Art of Doing Science and Engineering: Learning to Learn

Tags: #science #engineering #learning #mathematics #computer science #physics #artificial intelligence #systems thinking

Authors: Richard W. Hamming

Overview

This book, born from a graduate course I teach, is about instilling a style of thinking in future scientists and engineers, not just training them in technical details. The world is changing too rapidly to rely solely on what we’ve learned in the past. Instead, you must learn to learn, constantly adapting to new challenges and recognizing the enduring principles that underlie technological revolutions. I use examples from my career, particularly my time at Bell Labs, to illustrate key concepts. You’ll see how I approached problems, interacted with experts, recognized and seized opportunities, and learned from both successes and failures. I emphasize the importance of: understanding fundamental principles, recognizing the limitations of human intuition and expertise, questioning assumptions, seeking analogies, and having the courage to pursue important problems, even when the path is uncertain. We explore themes like the digital revolution, the limitations of artificial intelligence, the nature of mathematics, the strangeness of quantum mechanics, and the importance of systems thinking. Ultimately, my aim is to increase the probability that you, the reader, will become a significant contributor to society. The future is full of challenges, and those who are prepared to learn, adapt, and think creatively will be the ones who shape it.

Book Outline

1. Orientation

This book, born from my graduate course in engineering, is about instilling a style of thinking in budding scientists and engineers – preparing them for their future, not just my (or any teacher’s) past. The rapid pace of technological change demands we focus on learning to learn and recognizing the underlying principles that endure. The examples here, many from my time at Bell Labs, are less important than the methods of thought they illustrate.

Key concept: Teachers should prepare the student for the student’s future, not for the teacher’s past.

2. Foundations of the Digital (Discrete) Revolution

We are in the midst of a digital revolution, driven by the power of discrete signals to handle noise and the incredible advances in computing power. This has profound effects on both science and engineering. Engineering is becoming less about ‘what can we do?’ and more about ‘what do we want to do?’. The lines between science and engineering are blurring as computer simulations become central to both.

Key concept: In science if you know what you are doing you should not be doing it. In engineering if you do not know what you are doing you should not be doing it.

3. History of Computer—Hardware

The history of computing hardware shows a relentless drive toward faster speeds and increased storage. From pebbles and abacuses to slide rules and mechanical calculators, we progressed to vacuum tubes and transistors, ultimately reaching the era of integrated circuits. This growth follows an ‘S’ curve, suggesting we’re nearing saturation for the single processor von Neumann architecture and highlighting the importance of parallel processing.

Key concept: The purpose of computing is insight, not numbers.

4. History of Computer—Software

The evolution of computer software mirrors the hardware trajectory – from manual control and punched cards to symbolic programming and high-level languages like FORTRAN. Progress is often resisted, even by the professionals who would benefit most. Just as doctors don’t always follow their own health advice, programmers were initially reluctant to embrace tools like FORTRAN that ultimately made them far more productive. We moved from absolute to virtual machines, increasing the user’s ability to focus on the problem, not the machine’s intricacies.

Key concept: The shoe maker’s children go without shoes.

5. History of Computer Applications

The use of computers is rapidly expanding beyond scientific number crunching to encompass engineering, business, and entertainment. This expansion creates a need to handle massive amounts of data and raises challenging questions about artificial intelligence. But to understand the future of computing, we must look beyond linear projections of current trends and try to envision possibilities, guided by an understanding of fundamental forces, including human factors.

Key concept: No vision, not much of a future.

6. Artificial Intelligence—I

The question of whether machines can think, central to the field of artificial intelligence, is more accurately phrased as, “Can we write programs which will produce ‘thinking’ from a von Neumann type machine?”. While we’ve seen impressive feats in game playing and theorem proving, these successes often lead us to reinterpret the tasks as merely mechanical routines. True intelligence might involve not just what is done, but how it is done. This is a question that cannot be easily settled with current technology and definitions, but is nonetheless vital for our future.

Key concept: Can Machines Think?

7. Artificial Intelligence—II

Computers are not just about material progress, they are tools for expanding the human intellect. They are reshaping music composition, scientific experimentation, and even medical diagnosis. The real power lies in the combination of human and machine, each augmenting the other’s strengths. However, we must be wary of becoming overly reliant on machines and losing touch with the nuances of reality.

Key concept: The unexamined life is not worth living.

8. Artificial Intelligence—III

The debate over machine intelligence is often clouded by wishful thinking and a lack of clear definitions. To truly grasp the possibilities and limitations of AI, one must engage with the problem directly, trying to write programs that exhibit intelligent behavior and critically evaluating the results. Programming is in many ways more akin to novel writing than to engineering; both involve a creative component that cannot be easily reduced to a set of rules.

Key concept: Almost everyone who opens up a new field does not really understand it the way the followers do.

9. n-Dimensional Space

Thinking in n-dimensional space, where n is the number of independent parameters, is crucial for understanding complex systems and design problems. Our intuition often fails in higher dimensions. For instance, in n-dimensional space, a sphere’s volume is concentrated near its surface, suggesting optimal designs often lie at the edges of feasibility. Furthermore, almost all diagonal lines in n-dimensional space are nearly perpendicular to the coordinate axes.

Key concept: n-dimensions is indeed vast!

10. Coding Theory—I

Coding theory deals with the efficient representation of information for transmission or storage. Huffman coding provides a way to design variable-length codes that minimize the average message length, especially when symbol probabilities are highly uneven. The Kraft inequality establishes a limit on the possible lengths of code symbols for unique decodability.

Key concept: Huffman coding pays off when the probabilities of the symbols are very different, and does not pay off much when they are all rather equal.

11. Coding Theory—II

Beyond source encoding, we need channel encoding to deal with noise in communication or storage. Simple parity checks can detect single errors. For human-generated errors, weighted codes like those used in ISBN numbers are more effective. The design of efficient codes depends on understanding the nature of the errors and the statistics of the messages being sent.

Key concept: Rule: Huffman coding pays off when the probabilities of the symbols are very different, and does not pay off much when they are all rather equal.

12. Error Correcting Codes

Error correcting codes allow us to recover from transmission or storage errors. By arranging message bits in a hypercube and using parity checks, we can locate and correct single-bit errors. The design of such codes involves a trade-off between redundancy and error correction capability, and often single-error correction with double-error detection is a good balance.

Key concept: Luck favors the prepared mind.

13. Information Theory

Information Theory, while named for “information,” is really about surprise and uncertainty in streams of symbols. The amount of information gained from an event is inversely proportional to its probability. Shannon’s concept of information may not fully capture the human notion of information, but it is useful for designing efficient communication systems.

Key concept: This is the Cauchy functional equation, true for all p1and p2.

14. Digital Filters—I

Digital filters process streams of equally-spaced numbers, often representing sampled signals. Fourier analysis, employing complex exponentials, is well-suited for this task due to its relationship to time invariance, linearity, and the Nyquist sampling theorem. Aliasing, the phenomenon where higher frequencies are folded into lower ones during sampling, is a fundamental consequence of working with discrete signals.

Key concept: If you have either a time invariant system, or a linear system, then you should use the complex exponentials.

15. Digital Filters— II

Simple digital filters, like running averages, can be designed using least squares fitting. Designing more sophisticated filters often involves approximating ideal transfer functions with truncated Fourier series. The Gibbs phenomenon, an overshoot that occurs at discontinuities in least squares approximations, can be mitigated with windowing techniques like the Lanczos, von Hann, and Hamming windows.

Key concept: You have just seen a digital filter in action. The filter decomposes the input signal into all its frequencies, multiplies each frequency by its corresponding eigenvalue (the transfer function), and then adds all the terms together to give the output. The simple linear formula of the filter does all this!

16. Digital Filters—III

A systematic design process for non-recursive filters involves defining the desired filter shape, computing the corresponding Fourier coefficients, truncating the series, and applying a window function. The Kaiser window, a flexible family of window functions, allows for control over ripple height and transition width, offering a more systematic design approach than trial and error.

Key concept: I0(x) is the pure imaginary Bessel function of order 0. For computing it you will need comparatively few terms as there is an n! squared in the denominator and hence the series converges rapidly.

17. Digital Filters—IV

Recursive filters use feedback, incorporating past output values into the current output calculation. This allows for the implementation of integrators and other powerful operations, but also introduces the possibility of instability. The choice of filter design depends on the specific application, and often recursive filters offer advantages in terms of efficiency and complexity compared to non-recursive filters.

Key concept: Thus I have to this day the attitude stability in digital filters means “not exponential growth” from bounded inputs, but allows polynomial growth, and this is not the standard stability criterion derived from classical analog filters, where if it were not bounded you would melt things down —and anyway they had never really thought hard about integration as a filter process.

18. Simulation—I

Computer simulation is a powerful tool for answering “what if” questions, driven by the increasing power and decreasing cost of computers. Simulations can be cheaper, faster, and more accurate than real-world experiments. Crucially, simulations allow us to explore scenarios that are impossible or too dangerous to test in reality.

Key concept: The purpose of computing is insight, not numbers.

19. Simulation—II

While simulations are incredibly powerful, their reliability is paramount. We must critically evaluate the assumptions and limitations of our models, comparing the results with real-world data whenever possible. Furthermore, we must recognize the limitations of human judgment in both designing and interpreting simulations. The temptation to find what we want to find is strong, and we must guard against self-delusion.

Key concept: Computer based simulation is now in wide spread use to analyse system models and evaluate theoretical solutions to observed problems. Since important decisions must rely on simulation, it is essential that its validity be tested, and that its advocates be able to describe the level of authentic representation which they achieved.

20. Simulation—III

The ‘garbage in, garbage out’ principle (GIGO) does not always hold. Systems with feedback can be surprisingly robust to errors in the input data. Conversely, even with accurate input, simulations of unstable systems are prone to diverge from reality due to the accumulation of small errors. Understanding the stability of the system being simulated is crucial for assessing the reliability of the results.

Key concept: If the problem was important and properly posed then I could get some kind of a solution. Therefore, I must find the solution; I had no escape if I were to hold on to my pride.

21. Fiber Optics

Fiber optics is an example of a technology I watched unfold in my career. Understanding the fundamentals – the physics of light transmission, the advantages of high bandwidth, and the potential impact on communication infrastructure – allowed me to anticipate its importance and contribute to its development. The interplay of technological feasibility, economic viability, and social factors is crucial in determining which technologies ultimately succeed.

Key concept: “Luck favors the prepared mind.”

22. Computer Aided Instruction—CAI

Computer-aided instruction (CAI) is promising for rote learning and skill training, where conditioned responses are key. However, it’s less clear if computers can truly foster high-level thinking and creativity. The Hawthorne effect, where people perform better when they feel special attention is paid to them, complicates the evaluation of CAI’s effectiveness. Furthermore, the temptation to make learning ‘easier’ by simplifying material can backfire by hindering the development of deep understanding.

Key concept: What you learn from others you can use to follow; What you learn for yourself you can use to lead.

23. Mathematics

Mathematics is a powerful tool for clear thinking, but it is not an absolute truth, independent of human interpretation. While its theorems may seem eternally true, the underlying postulates and definitions evolve over time, driven by both practical needs and deeper understanding. The unreasonable effectiveness of mathematics lies in our ability to find analogies between its formal structures and aspects of reality. However, we must be wary of relying too heavily on the mathematical tools themselves, losing sight of the underlying assumptions and their limitations.

Key concept: “When rigor enters, meaning departs.”

24. Quantum Mechanics

Quantum Mechanics (QM) has revealed a universe far stranger than our classical intuition suggests. Its probabilistic nature, the wave-particle duality, and the phenomenon of non-local effects, all challenge our common sense notions of reality. While QM’s mathematical formalism has proven incredibly successful, we may never be able to truly “understand” it in the classical sense. This suggests that as we explore new frontiers of knowledge, we may encounter more and more phenomena that defy our intuition and require new modes of thought.

Key concept: Man is not a rational animal, he is a rationalizing animal.

25. Creativity

Creativity is often misunderstood as mere novelty or originality. True creativity involves generating valuable, novel ideas, though value is subjective and often only recognized in hindsight. The process of creativity often involves periods of intense focus, temporary abandonment, flashes of insight, and then refinement and revision. Preparing your mind by deeply understanding the fundamentals of your field, exploring analogies, and embracing challenges can increase the likelihood of creative breakthroughs.

Key concept: “Luck favors the prepared mind.”

26. Experts

Experts, while essential for progress, can also be a hindrance to innovation. They tend to be deeply invested in the current paradigm, often dismissing new ideas that challenge their beliefs or fall outside their area of expertise. Furthermore, the very process of becoming an expert can lead to a narrow focus, missing the bigger picture and the opportunities for breakthroughs that lie at the edges of the field.

Key concept: “If an expert says something can be done he is probably correct, but if he says it is impossible then consider getting another opinion.”

27. Unreliable Data

Data is often far less reliable than we assume, and its accuracy should not be confused with its relevance to our goals. Life testing, based on accelerated aging and extrapolation, is inherently uncertain. Economic and social data is often biased, incomplete, or based on shifting definitions. When gathering data, focus on careful design, recognize the limitations of large-scale data collection, and be wary of the influence of phrasing and context in questionnaires.

Key concept: Small samples carefully taken are better than large samples poorly done.

28. Systems Engineering

Systems engineering is about understanding how the parts of a system interact to achieve the desired overall performance. Optimizing individual components in isolation often leads to suboptimal system behavior. Furthermore, good system design anticipates change, allowing for flexibility and graceful degradation when overloaded. The solution to a systems engineering problem is rarely final, but rather evolves over time as our understanding of the system deepens.

Key concept: If you optimize the components you will probably ruin the system performance.

29. You Get What You Measure

Measurement systems have profound effects on the systems they measure. The choice of measurement scales, the range of ratings used, and the emphasis on hard versus soft metrics all influence behavior and outcomes. Rating systems can inadvertently incentivize risk aversion or narrow focus, filtering out the very people needed for innovation and leadership at higher levels. When designing measurement systems, carefully consider the unintended consequences and strive to align them with the desired goals.

Key concept: You get what you measure.

30. You and Your Research

To make a significant impact in your career, focus on important problems, cultivate confidence and a drive for excellence, and learn to tolerate ambiguity, embracing both belief and disbelief. Be willing to step back from the daily grind and reflect on the larger questions that will shape the future of your field. Remember that luck favors the prepared mind, and while you cannot control everything, you can control how you prepare yourself to seize opportunities when they arise.

Key concept: “The unexamined life is not worth living.”

Essential Questions

1. How can we best prepare for a future of rapid technological change, where the knowledge we acquire today may be obsolete tomorrow?

The rapid pace of technological advancement demands we focus on constantly learning and adapting. By mastering the fundamentals of our fields and developing a mindset for independent thought and problem-solving, we can better navigate the ever-changing landscape of knowledge. This involves recognizing that what we were taught may become obsolete and embracing a willingness to learn new concepts and approaches throughout our careers. The book emphasizes a style of active learning, encouraging readers to critically examine information, question assumptions, and actively pursue deeper understanding.

2. What are the fundamental differences between human intelligence and artificial intelligence, and how can we best leverage the strengths of both?

While computers excel in speed, accuracy, and reliability, they lack the human capacity for genuine understanding and creativity. The book explores this distinction through examples of computer programs that exhibit seemingly intelligent behavior, like playing games or proving theorems. Hamming argues that while these programs can achieve impressive results, they often rely on brute-force computation or clever programming tricks rather than true insight. The question of whether machines can think remains open, highlighting the unique qualities of human intelligence and the need to leverage the strengths of both humans and computers.

3. How can we effectively leverage expertise while remaining open to new ideas and avoiding the pitfalls of blind deference to authority?

Hamming argues that experts, while essential for progress, can also hinder innovation by clinging to outdated paradigms and dismissing new ideas that challenge their established beliefs. To navigate this, we must cultivate a balance between respecting expertise and maintaining a healthy skepticism towards it. It’s important to question assumptions, recognize the limitations of specialized knowledge, and be open to ideas from outside our own fields. The book highlights examples of how paradigm shifts in science and technology often arise from outsiders who bring fresh perspectives and challenge conventional thinking.

4. How do our choices of what to measure, and how to measure it, influence outcomes and shape the systems we are trying to understand or improve?

While accurate measurement is important, the choice of what to measure has profound effects on the system being measured. Rating systems, performance metrics, and even scientific experiments can inadvertently incentivize behaviors that are detrimental to the overall goals. Hamming emphasizes the need to carefully consider the unintended consequences of measurement, ensuring that what we measure aligns with what we truly value. He highlights examples of how poorly chosen metrics can lead to short-term focus, risk aversion, and a narrowness of vision, hindering both individual and organizational progress.

5. How can we apply systems thinking to approach complex problems, considering the interconnectedness of components and the need for adaptability in the face of change?

The book champions a holistic approach to problem solving, emphasizing the interconnectedness of systems and the need to consider the larger picture when making decisions. Optimizing individual components in isolation can lead to suboptimal system performance. Effective system design involves anticipating change, allowing for flexibility, and ensuring graceful degradation under overload conditions. The book highlights the importance of understanding the fundamental dynamics of the system, recognizing that solutions are rarely final, and embracing a process of continuous evolution and improvement.

Key Takeaways

1. Optimizing individual components in isolation often leads to suboptimal system performance.

This is a recurring theme throughout the book, illustrated with examples from engineering, computer science, and even personal experience. Hamming argues that a narrow focus on optimizing individual parts often leads to suboptimal results for the system as a whole. A good systems engineer must consider the interactions between components, the overall goals, and the constraints of the environment.

Practical Application:

When designing an AI system for, say, image recognition, it’s tempting to focus solely on achieving state-of-the-art accuracy on benchmark datasets. However, real-world performance might be significantly lower due to factors not captured in the benchmarks. A more systems-oriented approach would involve considering factors like: computational efficiency on target hardware, robustness to variations in input data, and explainability of the AI’s decisions. These might require trade-offs with pure accuracy but would lead to a more practical and valuable system.

2. A deep understanding of the problem to be solved is crucial for choosing the right tools and achieving meaningful results.

This emphasizes the importance of understanding the context in which technology will be applied. A deep understanding of the problem, the users, and the broader environment leads to more effective and valuable solutions. This requires going beyond the immediate technical details and considering the larger picture.

Practical Application:

Imagine you’re leading a team developing a new AI-powered chatbot. The temptation is to focus on implementing the latest deep learning models for natural language processing. However, Hamming’s advice suggests you should first carefully define the problem the chatbot is supposed to solve, considering factors like: the target audience, the desired user experience, and the ethical implications of the technology. This deep understanding of the problem will guide the choice of technology and lead to a more successful and impactful solution.

3. The ability to learn and adapt is essential for success in a rapidly changing technological landscape.

The book highlights the rapid pace of technological change and the need to constantly adapt. Hamming encourages readers to embrace a mindset of continuous learning, exploring new fields, questioning assumptions, and seeking new perspectives. This is crucial for staying relevant and making significant contributions in a rapidly evolving world.

Practical Application:

As an AI product engineer, you should actively cultivate a mindset of continuous learning. Stay abreast of new research in AI, but also explore adjacent fields like cognitive science, neuroscience, and philosophy. These can provide fresh perspectives and insights into the nature of intelligence and the potential of AI. Furthermore, consider the social and ethical implications of AI, as these will increasingly shape its development and application.

Suggested Deep Dive

Chapter: n-Dimensional Space

The chapter on n-dimensional space offers a counterintuitive view of higher dimensional spaces, crucial for understanding complex systems and optimization problems. It challenges the reader to rethink their intuition and provides a foundation for grasping the concepts behind many modern AI techniques that operate in high-dimensional spaces.

Memorable Quotes

Preface. 7

Teachers should prepare the student for the student’s future, not for the teacher’s past.

History of Computers—Hardware. 31

The purpose of computing is insight, not numbers.

Preface. 8

Luck favors the prepared mind.

Orientation. 2

Education is what, when, and why to do things, Training is how to do it.

Systems Engineering. 196

If you optimize the components you will probably ruin the system performance.

Comparative Analysis

Hamming’s book shares thematic similarities with classic works like Claude Shannon’s “A Mathematical Theory of Communication” and Thomas Kuhn’s “The Structure of Scientific Revolutions.” Shannon’s work lays the foundation for information theory, which Hamming extends by focusing on its practical applications in coding and digital filter design. Kuhn’s exploration of paradigm shifts in science resonates with Hamming’s emphasis on the limitations of expertise and the importance of recognizing when the ‘rules’ of the game have changed. However, Hamming’s book is unique in its focus on personal style and the development of a successful scientific mindset. He draws heavily on personal anecdotes and observations, offering practical advice on topics like learning new fields, recognizing opportunities, and interacting with experts. This gives the book a personal touch and a pragmatic focus that sets it apart from more abstract or theoretical works in the field.

Reflection

Hamming’s book is a powerful call to action for anyone aspiring to make a significant contribution in their field. It’s a reminder that technical proficiency is not enough; we must also cultivate a mindset for independent thought, creativity, and a willingness to challenge the status quo. While his emphasis on personal anecdotes can feel a bit self-congratulatory at times, the underlying message is both inspiring and thought-provoking. However, his dismissal of social sciences and his skepticism towards artificial intelligence may seem outdated in light of recent progress in those fields. His views on the limitations of human intelligence and the potential for machines to surpass us warrant further scrutiny. Despite its flaws, the book’s core message – the importance of lifelong learning, pursuing important problems, and thinking in a systems-oriented way – remains highly relevant in today’s rapidly changing world. It’s a valuable read for anyone who wants to navigate the complexities of technological progress and make a meaningful impact.

Flashcards

Explain the ‘drunken sailor’ analogy as it relates to having a vision in one’s career.

A drunken sailor who staggers to the left or right with n independent random steps will, on average, end up about the square root of n steps from the origin. But if there is a pretty girl in one direction, then he will go a distance proportional to n.

What is the difference between education and training?

Education is what, when, and why to do things. Training is how to do it.

What is the first rule of systems engineering?

Optimizing individual components in isolation can lead to suboptimal system performance.

What is Hamming’s rule about measurement accuracy?

90% of the time the next independent measurement will fall outside the previous 90% confidence limits.

How is information measured in Information Theory?

The amount of information gained from an event is inversely proportional to its probability.

What is aliasing in digital signal processing?

Aliasing is the phenomenon where higher frequencies are folded into lower ones during sampling.

What is Huffman Coding?

A method of designing variable-length codes that minimize average message length, especially effective when symbol probabilities are highly uneven.

What is the Kraft inequality?

A limit on the possible lengths of code symbols for unique decodability.